Unless otherwise indicated herein, the materials described in this section are not prior art to the claims in this application and are not admitted to be prior art by inclusion in this section.
Many environments can be explored by robots, such as undersea, outer space, and hazardous environments. In some of these environments, a robotic device can be controlled by a human operator receiving video and/or audio information from the robotic device. Human control of the robotic device combines the experience and awareness of a human being with the precision, repeatability and accuracy of machines by providing primary control of robotic motion.
A challenge can arise when trying to provide a virtual environment to the human operator that is realistic, immersive representation of a remote environment where a robot is operating, so that the human operator can effectively and easily control a robot via interactions with the virtual environment. The human operator should receive as much information about the remote environment as possible in an intuitive manner so the operator's focus is not distracted from the remote task at hand. Feedback qualities are critical to the effectiveness of a remote operation user interface used to facilitate these interactions with the virtual environment. Then, the human operator can make proper decisions to efficiently complete the task with minimal physical and mental effort.
A combination of feedback quantities beyond simple monocular vision can assist the operator by overlaying sensory information over data captured from the remote scene using virtual haptic fixtures. For example, virtual haptic fixtures and three-dimensional (3D) mapping techniques used to provide the virtual environment can enhance task efficiency and awareness. In particular, one or more haptic guidance fixtures can be provided using haptic rendering techniques that translate forces in a virtual environment to a physical device that can provide touch-based, a.k.a. haptic, feedback to a user of the haptic rendering device. The haptic rendering device can be controlled using a “Haptic Interaction Point” (HIP) in the virtual environment, which performs a similar function for the haptic rendering device as a mouse pointer does for a computer mouse. Ideally, the HIP should not be able to penetrate virtual environment objects.
Along with providing haptic feedback about the virtual environment, objects in the virtual environment are often represented as a collection of polygons, such as triangles, that can be operated upon using a haptic rendering device. In some cases, 3D mapping can be used to enhance visual feedback provided to the human operator about the remote environment.
In recent years, inexpensive Red-Green-Blue-Depth (RGB-D) cameras can provide non-contact means of collecting geometrical and boundary information of an observed (remote) environment in real time and providing the collected information as point cloud data to a remote computing device providing the virtual environment. The point cloud data to can be useful for providing haptic interaction with the virtual environment, calculating grasper configurations, and for other tasks. Several technologies, such as time-of-flight and structured light, have been used in different commercially available depth cameras
For example, an RGB-D camera can gather color and geometric data from the remote environment. This color and geometric data can be used to generate a virtual environment where the human operator is presented with a monocular color video stream and a 3D-mapping voxel representation of the remote scene. In some cases, the human operator can interact with and/or place virtual haptic fixture(s) to complete a remote manipulation task, such as grasping and/or turning an object in the remote environment.
Techniques for six degree-of-freedom haptic rendering in virtual environments consisting of polygons and/or voxels (volume pixels) have been specified. These efforts are typically divided into direct rendering and virtual coupling methods where the latter can further be subdivided into penalty-, impulse- and constraint-based methods. The simplest 6-DOF haptic rendering method is the direct method, where the virtual tool perfectly matches the configuration of the haptic rendering device. The force sent to user is directly based on the amount of penetration in the virtual environment. Unfortunately the direct method suffers from problems with “pop through”. Pop through is an artifact that arises when the rendering algorithm erroneously penetrates a thin surface.
In virtual coupling methods, a virtual coupling, or connection, between the haptic rendering device and the virtual tool is utilized. In this method, the force on the haptic rendering device is simply calculated as a spring between the virtual tool, referred to also as “god-object”, “proxy” or “IHIP”, and the configuration of the haptic rendering device. 6-DOF rendering methods using virtual couplings rely on rigid body simulations, since the virtual tool has to be simulated as a 6-DOF object, as compared to 3-DOF rendering where the rotational component can be ignored. In penalty-based methods, the configuration (position and rotation) of the virtual tool is calculated using penalty forces based on the tool's penetration depth into objects, similar to how penalty-costs are used in traditional optimization. These penalty forces are then integrated to produce the motion of the virtual tool. This method results in a virtual tool that actually penetrates objects in the environment. Fortunately this penetration is typically very small.
For impulse-based dynamics methods, a virtual object is moved by a series of impulses upon contact/collision (rather than forces based on penetration depth). In constraint-based methods, the virtual tool moves into contact with the environment but (ideally) never violates constraints imposed by the environment.